A case study of pragmatic measure development of the Treatment Integrity for Elementary Settings self-report measure for teachers.

Introduction: Due to usability, feasibility, and acceptability concerns, observational treatment fidelity measures are often challenging to deploy in schools. Teacher self-report fidelity measures with specific design features might address some of these barriers. This case study outlines a community-engaged, iterative process to adapt the observational Treatment Integrity for Elementary Settings (TIES-O) to a teacher self-report version designed to assess the use of practices to support children's social-emotional competencies in elementary classrooms. Method: Cognitive walkthrough interviews were conducted with teachers to improve the usability of the teacher self-report measure, called the Treatment Integrity for Elementary Schools-Teacher Report (TIES-T). Qualitative content analysis was used to extract themes from the interviews and inform changes to the measure. Results: Increasing clarity and interactive elements in the measure training were the dominant themes, but suggestions for the measure format and jargon were also suggested. Conclusion: The suggested changes resulted in a brief measure, training, and feedback system designed to support the teacher's use of practices to support children's social-emotional competencies in elementary classrooms. Future research with the TIES-T will examine the score reliability and validity of the measure.

Collecting observational data in schools is challenging, so developing teacher self-report measures and involving teachers in the design process is important to help make them easier to use. This paper reports on the development of a teacher self-report measure designed to collect information about the instructional practices teachers deliver to promote positive student behavior.

Examining the Role of Student Responsiveness in Treatment Effects of a Tier 2 Program Targeting Reductions in Problem Behavior.

Student responsiveness's role in promoting intervention outcomes for students who exhibit problem behavior is understudied. Due to the relational nature of many interventions delivered by teachers that target social, emotional, or behavioral outcomes of students in classrooms, it is essential to assess how responsive students are to teachers' attempts to engage them in the intervention, particularly for students with problem behaviors that may impede teachers' attempts to engage these students in intervention effectively. In the current study, we combine samples from four randomized controlled trials to examine the relationship between student outcomes and teacher attempts to deliver BEST in CLASS, a Tier 2 intervention, via student responsiveness. Delivery of BEST in CLASS and student responsiveness were assessed through direct observations and teachers' reported measures. Results suggest that teacher adherence and competence in delivering BEST in CLASS practices was associated with reductions in problem behavior from pretest to post-test via student responsiveness. Limitations of the current study and implications for future research are discussed.

Study Protocol: Multi-level Determinants of Implementation and Sustainment in the Education Sector.

Evidence-based programs (EBPs) delivered in elementary schools show great promise in reducing risk for emotional and behavioral disorders (EBDs). However, efforts to sustain EBPs in school face barriers. Improving EBP sustainment thus represents a priority, but little research exists to inform the development of sustainment strategies. To address this gap, the Sustaining Evidenced-Based Innovations through Multi-level Implementation Constructs (SEISMIC) project will: (a) Determine if malleable individual, intervention, and organizational factors predict EBP treatment fidelity and modifications during implementation, sustainment, or both; (b) Assess the impact of EBP fidelity and modifications on child outcomes during implementation and sustainment; and (c) Explore the mechanisms through which individual, intervention, and organizational factors influence sustainment outcomes. This protocol paper describes SEISMIC, which builds upon a federally-funded RCT evaluating BEST in CLASS, a teacher-delivered program for K-3rd grade children at risk for EBDs. The sample will include 96 teachers, 384 children, and 12 elementary schools. A multi-level, interrupted time series design will be used to examine the relationship between baseline factors, treatment fidelity, modifications, and child outcomes, followed by a mixed-method approach to elucidate the mechanisms that influence sustainment outcomes. Findings will be used to create a strategy to improve EBP sustainment in schools.

Sustaining BEST in CLASS: Teacher-Reported Evidence-Based Practice Use with Students at Risk for Emotional and Behavioral Disorders Amidst the COVID-19 Pandemic.

There is growing evidence of the efficacy of evidence-based interventions in improving the academic and social outcomes of children who exhibit challenging behaviors during program implementation periods. However, less is known about the extent to which practices learned as part of these interventions are sustained after these projects end, when funding is paused temporarily, and in less-than-ideal conditions. This study used qualitative methods to investigate whether teachers previously trained in the BEST in CLASS-Elementary intervention continued to use the program's evidence-based practices with students 1-2 years after completing the program and in the context of the COVID-19 pandemic. It also examined teachers' perceptions of the impact of practice use on students' academic and social outcomes. Thirteen BEST in CLASS-Elementary teachers from elementary schools in two southeastern states in the USA where the program was implemented completed semi-structured interviews on the topic. Data were coded thematically, and the results indicated that over 50% of teachers reported using "rules," "supportive relationships," and "praise" frequently with their students. However, "precorrection" and "opportunities to respond" were reportedly used less often. Teachers also perceived that their use of these evidence-based practices was linked to increases in their students' academic engagement and academic performance and knowledge, improvements in students' behaviors, their relationships with teachers, and general comfort and self-confidence. The discussion highlights modality-specific patterns noted in the results that might influence sustainment and the implication of these findings for interventions and programs aimed at promoting positive behavioral outcomes for early elementary school students.

Development and Initial Psychometrics of a Generic Treatment Integrity Measure Designed to Assess Practice Elements Targeting Social, Emotional, and Behavioral Outcomes in Early Childhood Settings.

Though treatment integrity measurement is important for research intended to promote social and behavioral outcomes of children at risk for emotional and behavioral disorders (EBDs) in early childhood settings, measurement gaps exist in the field. This paper reports on the development and preliminary psychometric assessment of the treatment integrity measure for early childhood settings (TIMECS), an observational measure designed to address existing measurement gaps related to treatment integrity with tier 2 interventions in the early childhood field. To assess the preliminary score reliability (interrater) and validity (construct, discriminant) of the TIMECS, live observations (N = 650) in early childhood classrooms from 54 teachers (92.6% female, 7.4% male; 61.1% White) and 91 children (M age = 4.53 years, SD = .44; 45.1% female, 54.9% male; 45.1% Black) at risk for EBDs were scored by 12 coders using the TIMECS and an observational measure designed to assess teacher-child interactions. Teachers also self-reported on the quality of the teacher-child relationship. Interrater reliability (intraclass correlation coefficients, ICC [2,2]) for the quantity (i.e., adherence) item scores had a mean of .81 (SD = .07; range from .68 to .95), and the quality (i.e., competence) item scores had a mean of .69 (SD = .08; range from .52 to .80). Scores on the TIMECS Quantity and Quality items and scales showed evidence of construct validity, with the magnitude of the correlations suggesting that the quantity and quality items assess distinct components of treatment integrity. A TIMECS quantity scale also showed promise for intervention evaluation research by discriminating between teachers who had and had not been trained in a specific evidence-based intervention targeting social and behavioral skills in early childhood. The findings support the potential of the TIMECS to assess treatment integrity of teacher-delivered practices designed to address child social and behavioral outcomes of children at risk for EBDs in early childhood settings.

Evaluation of the Olweus Bullying Prevention Program in US Urban Middle Schools Using a Multiple Baseline Experimental Design.

We evaluated the impact of the Olweus Bullying Prevention Program (OBPP) in an 8-year study in urban middle schools that served primarily African American students living in low-income areas. Participants included 2755 students and 242 teachers. We evaluated the OBPP with a multiple-baseline experimental design where the order and intervention start time was randomly assigned for each school. We assessed the frequency of bullying behaviors and experiences including physical, relational, and verbal aggression and victimization using teacher ratings of student behavior and student-reported data, as well as cyber aggression and victimization and school climate measures using student-reported data. For teacher ratings of student behavior, we found significant main effects across all subtypes of aggression and victimization, with some variability in the timing of effects. The pattern of findings showed delayed intervention effects for boys and a weaker impact of the OBPP on 6th graders. We found main effects for student-reported cyber aggression and victimization, relational aggression, and a composite of physical, verbal, and relational victimization. Decreases in victimization emerged in the 1st or 2nd year of intervention, and reductions in aggression emerged during the 3rd year. Across all findings, once intervention effects emerged, they remained significant in subsequent intervention years. The OBPP resulted in significant decreases in student- and teacher-reported aggression and victimization. However, this intervention had limited impact on general areas of school climate including teacher support, positive peer interactions, and school safety. Overall, the findings offer important prevention and research implications.

Evaluation of the Olweus Bully Prevention Program in an Urban School System in the USA.

This study evaluated the Olweus Bully Prevention Program (OBPP) in urban middle schools serving a mostly African American student population. Participants were 1791 students from three communities with high rates of crime and poverty. We evaluated the impact of the OBPP using a multiple-baseline experimental design in which we randomized the order and timing of intervention activities across three schools. We assessed the frequency of violence and victimization using self-report and teachers' ratings of students collected every 3 months over 5 years. Initiation of the OBPP was associated with reductions in teachers' ratings of students' frequency of aggression, with effects emerging in different years of implementation for different forms of aggression. Whereas reductions in teachers' ratings of students' verbal and relational aggression and victimization were evident during the second implementation year, reductions in physical aggression did not appear until the third year. Effects were consistent across gender and schools, with variability across grades for relational and verbal aggression and victimization. In contrast, there were no intervention effects on students' reports of their behavior. Positive outcomes for teachers', but not students' ratings, suggest the intervention's effects may have been limited to the school context. Variation in when effects emerged across outcomes suggests that changes in physical aggression may require more sustained intervention efforts. The intervention was also associated with increases in teachers' concerns about school safety problems, which may indicate that teachers were more attuned to recognizing problem behaviors following exposure to the OBPP.

Factors Associated with Teacher Delivery of a Classroom-Based Tier 2 Prevention Program.

Teachers sometimes struggle to deliver evidence-based programs designed to prevent and ameliorate chronic problem behaviors of young children with integrity. Identifying factors associated with variations in the quantity and quality of delivery is thus an important goal for the field. This study investigated factors associated with teacher treatment integrity of BEST in CLASS, a tier-2 prevention program designed for young children at risk for developing emotional/behavioral disorders. Ninety-two early childhood teachers and 231 young children at-risk for emotional/behavioral disorders participated in the study. Latent growth curve analyses indicated that both adherence and competence of delivery increased across six observed time points. Results suggest that teacher education and initial levels of classroom quality may be important factors to consider when teachers deliver tier-2 (i.e., targeted to children who are not responsive to universal or tier-1 programming) prevention programs in early childhood settings. Teachers with higher levels of education delivered the program with more adherence and competence initially. Teachers with higher initial scores on the Emotional Support subscale of the Classroom Assessment Scoring System (CLASS) delivered the program with more competence initially and exhibited higher growth in both adherence and competence of delivery across time. Teachers with higher initial scores on the Classroom Organization subscale of the CLASS exhibited lower growth in adherence across time. Contrary to hypotheses, teacher self-efficacy did not predict adherence, and teachers who reported higher initial levels of Student Engagement self-efficacy exhibited lower growth in competence of delivery. Results are discussed in relation to teacher delivery of evidence-based programs in early childhood classrooms.

Identifying Common Practice Elements to Improve Social, Emotional, and Behavioral Outcomes of Young Children in Early Childhood Classrooms.

Educators are increasingly being encouraged to implement evidence-based interventions and practices to address the social, emotional, and behavioral needs of young children who exhibit problem behavior in early childhood settings. Given the nature of social-emotional learning during the early childhood years and the lack of a common set of core evidence-based practices within the early childhood literature, selection of instructional practices that foster positive social, emotional, and behavioral outcomes for children in early childhood settings can be difficult. The purpose of this paper is to report findings from a study designed to identify common practice elements found in comprehensive intervention models (i.e., manualized interventions that include a number of components) or discrete practices (i.e., a specific behavior or action) designed to target social, emotional, and behavioral learning of young children who exhibit problem behavior. We conducted a systematic review of early childhood classroom interventions that had been evaluated in randomized group designs, quasi-experimental designs, and single-case experimental designs. A total of 49 published articles were identified, and an iterative process was used to identify common practice elements. The practice elements were subsequently reviewed by experts in social-emotional and behavioral interventions for young children. Twenty-four practice elements were identified and classified into content (the goal or general principle that guides a practice element) and delivery (the way in which a teacher provides instruction to the child) categories. We discuss implications that the identification of these practice elements found in the early childhood literature has for efforts to implement models and practices.

Measuring teacher implementation in delivery of a bullying prevention program: the impact of instructional and procedural adherence and competence on student responsiveness.

Although there is evidence that school-based prevention programs can produce positive effects on students' academic and behavioral functioning, the ability of teachers to sustain high-quality implementation remains an open and vexing question. Because teachers are often the intervention agents in school-based prevention programs, assessing both their adherence to program procedures and their competence in program delivery is critical for ensuring student responsiveness to prevention programs, which in turn may impact their efficacy. The current study assessed treatment fidelity of implementation of the Olweus' Bullying Prevention Program (OBPP) in two urban middle schools. Trained observers completed 280 observations of teachers' delivery of the class meeting component of the OBPP and rated teachers' instructional and procedural adherence and competence of delivery and students' responsiveness. Analyses using multilevel modeling indicated that competence of delivery was significantly related to student responsiveness above and beyond teacher instructional behavior adherence, such that class meetings conducted with higher instructional adherence and procedural competence resulted in higher student responsiveness to the program after controlling for the clustered nature of teachers, and several observation-level and teacher-level covariates. This study highlights the need for strategies to increase teacher use of effective instructional practices and competence with program procedures to enhance the efficacy of prevention programming in schools.